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SIMPLE LIE GROUPS 
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Abstract. We show that a surface group of high genus contained in 
a classical simple Lie group can be deformed to become Zariski dense, 
unless the Lie group is SU{p,q) (resp. SO*{2n), n odd) and the sur- 
face group is maximal in some S{U{p,p) x U{q — p)) C SU{p, q) (resp. 
SO* {2n—2)xSO{2) C SO* {2n)). This is a converse, for classical groups, 
to a rigidity result of S. Bradlow, O. Garcia-Prada and P. Gothen. 



Dedicated to Lionel Berard-Bergery, for his 65th birthday 
1. Introduction 

Free groups are obviously flexible. In particular, a generic free subgroup 
in a real algebraic group is Zariski dense. This already fails for surface 
groups, although they are very flexible from other points of view. The first 
evidence came from the following result by D. Toledo. 

Theorem 1.1. (D. Toledo, 1979, 1989, [15]). Let T be a discrete cocompact 
subgroup of SU (1,1). Map SU{l,l) as a 2 x 2 block in SU{l,n), n > 2. 
Then every neighboring homomorphism F — > SU{l,n) is contained in a 
conjugate of S{U{1,1) xU{n-l)). 

In fact, Toledo obtained a stronger, global result: a characterization of 
surface subgroups of S(U{1,1) x U{n — 1)) among surface subgroups of 
SU{l,n) by the value of a characteristic class known as Toledo's invariant, 
which we now define. 

Let X be a Hermitian symmetric space, with Kahler form Q (the met- 
ric is normalized so that the minimal sectional curvature equals —1). Let 
S be a closed surface of negative Euler characteristic, let T = 7ri(S) act 
isometrically on X. Pick a smooth equivariant map f : T, X. 

Definition 1.2. Define the Toledo invariant of the action p :T Lsom{X) 
by 



Then 

(1) Tp depends continuously on p. 



H. Kim gratefully acknowledges the partial support of NRF grant ((ROl-2008-000- 
10052-0) and a warm support of IHES during his stay. 
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(2) There exists £x ^ Q such that Tp G £x^- 

(3) \Tp\ < |x(S)|rank(X). 

Inequahty (3), known as the Milnor-Wood inequality for actions on Hermit- 
ian symmetric spaces, is due to J. Milnor, [13], V. Turaev, [16], A. Domic 
and D. Toledo, [C], J.-L. Clerc and B. 0rsted, [5]. 

Definition 1.3. Actions p such that \Tp\ = |x(S)|rank(X) are called maxi- 
mal representations. 

The following result generalizes Theorem 1.1. 

Theorem 1.4. (L. Hernandez Lamoneda, [11], S. Bradlow, O. Garcfa- 
Prada, P. Gothen, [1], [2]). Maximal reductive representations of T to 
SU{p,q), p < q, can be conjugated into S{U{p,p) x U{q — p))- Maximal 
reductive representations of T to SO*{2n), n odd, can he conjugated into 
SO*{2n-2) X 50(2). 

In turn. Theorem 1.4 is a special case of a more general result. 

Definition 1.5. Say a Hermitian symmetric space is of tube type if it can 
be realized as a domain in C" of the form M."" + iC where C C M" is a proper 
open cone. 

Example 1.6. Siegel's upper half spaces and Grassmannians with isometry 
groups P0{2,q) are of tube type. 

The Grassmannian 'Dp^g, p < q, with isometry group PU{p,q) is of tube 
type iffp = q. 

The Grassmannian Qn with isometry group SO*{2n) is of tube type iff n 
is even. 

The exceptional Hermitian symmetric space of dimension 27 is of tube 
type, the other one (of dimension 16) is not. 

Products of tube type spaces are of tube type, so polydisks are of tube type. 

Remark 1.7. All maximal tube type subsymmetric spaces in a Hermitian 
symmetric space are conjugate. For instance, the maximal tube type sub- 
symmetric space in Vp^g is 'Dp^p. The maximal tube type subsymmetric space 
in Q2n+i is G2n- 

Theorem 1.8. (Burger, lozzi, Wienhard, [3]). Let T be a closed surface 
group and X a Hermitian symmetric space. Every maximal representation 
T Isom{X) stabilizes a maximal tube type subsymmetric space Y . Con- 
versely, for every tube type Hermitian symmetric space X, Isom{X) admits 
Zariski dense maximal surface subgroups. 

1.1. Results. Our main result is a converse of Theorem 1.4 (i.e. Theorem 
1.8 for classical simple Lie groups). 

Theorem 1. Let G be a classical real Lie group, i.e. a real form of SL{n, C), 
0(n,C) or 5p(n,C). Let T be the fundamental group of a closed surface of 
genus > 2dim(G)^. A homomorphism cj) : T ^ G can be approximated by 
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Zariski dense representations, unless the symmetric space X of G is Her- 
mitian and not of tube type, and (p is maximal. 

In other words, the exceptions are G = SU{p,q), q > p and (j){T) is con- 
tained in a conjugate of S{U {p,p) x U{q—p)) C SU (p, q), or G = S0*{2n), n 
odd, and (/'(F) is contained in a conjugate of SO* {2n—2)x S0{2) C S0*{2n). 

The genus assumption is probably unnecessary, but we are unable to 
remove it. 

Question 1.9. Does Theorem 1 extend to exceptional simple Lie groups ? 

Since flexibility easily holds in compact and complex groups (see Propo- 
sition 4.3), and the rank one example F^'^^ is treated in [12], there remains 
10 cases [E^, E^^^, E^^^, E^, Ej^, 2^ E^ E^^^, , G^) with at least 
one rigidity case (Eq^^). 

1.2. Scheme of proof. The proof relies on 

• a necessary and sufficient condition for flexibility from [12], see The- 
orem 2.2 below; 

• tools from Burger, lozzi and Wienhard's theory of tight maps be- 
tween Hermitian symmetric spaces, [4]; 

• a detailed analysis of centers of centralizers of reductive subgroups 
of classical simple Lie groups. 

This last analysis is performed in a rather bare handed manner, based on bi- 
linear and sesquilinear algebra. This is where exceptional simple Lie groups 
elude us. 

1.3. Plan of the paper. Section 2 recalls the needed result from [12]. 
Section 3 proves relevant consequences of the theory of tight maps. Section 
4 provides a description of classical real simple Lie groups as fixed points 
of involutions which helps in computing root space decompositions in the 
complexified Lie algebra. This is done in section 5 for s[(n, C) and in section 
6 for so(n, C) and sp(?i,C). The method consists in first computing the 
root space decomposition in the standard representation of the complexified 
Lie algebra, and deducing the decomposition in the adjoint representation. 
The theory of tight homomorphisms allows to exclude balancedness (with 
exceptions), first for real forms of SL{n,C) in section 7, then for real forms 
of SO{n, C) and Sp{n, C) in section 8. Theorem 1 is proven in section 9. 

1.4. Acknowledgements. Many thanks to Marc Burger and Jean-Louis 
Clerc, who explained us tight maps and bounded symmetric domains. 

2. Flexibility criterion 

As far as the flexibility of a homomorphism (j) is concerned, a key role 
is played by the center of the centralizer of the image of (/>. It splits the 
complexified Lie algebra of G into root spaces q\. When the root A is pure 
imaginary, q\ carries a natural nondegenerate sesquilinear form defined as 
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follows. Let {X, X') X ■ X' denote the Killing form on g (g) C. Then the 
sesquilinear form 



is nondegenerate on g^. Let denote the imaginary part of sx. It is a 
symplectic form on qx viewed as a real vectorspace. The representation of 
r on Qx gives rise to a homomorphism F — ?> ^^(gA^^A), an isometric action 
on the Siegel domain, and thus a Toledo invariant Tx- 

Definition 2.1. Let i C Q be a torus, centralized by a homomorphism <j) '■ 
T ^ G. Among the roots of the adjoint action ofi on g, let P be the subset of 
pure imaginary roots A such that 2Tx = —x(X)d^^c{9x) , i-^- the symplectic 
r action on gx is maximal with positive Toledo invariant. Say t is balanced 
with respect to (p if belongs to the interior of the sum of the convex hull 
of the imaginary parts of elements of P and the linear span of the real and 
imaginary parts of roots not in ±P. 

Here is a necessary and sufficient condition for flexibility, for surface 
groups of sufficiently large genus. 

Theorem 2.2. ([12], Theorem 3). Let G be a semisimple real algebraic 
group. Let T be the fundamental group of a closed surface of genus > 
2dim(G)^. Let cp : T ^ G be a homomorphism with reductive Zariski clo- 
sure. Then (p is flexible if and only if c, the center of the centralizer of cj){T), 
is balanced with respect to (j). 

The proof of Theorem 1 will rely on this criterion: we shall describe 
centers of centralizers of reductive subgroups and their roots and pile up 
restrictions on the set P that make non balancedness exceptional. The 
reduction from nonreductive to reductive homomorphisms will be explained 
in section 9. 



We collect in this section properties related to maximality of represen- 
tations. We start with elementary facts. Deeper results will follow from 
tightness theory. 

3.1. Preservation of maximafity. We shall need that certain embeddings 
of Lie groups preserve maximality. 

Definition 3.1. Let p : H ^ G be a homomorphism between reductive 
Hermitian groups and F -.Y X denote an equivariant totally geodesic map 
between the corresponding Hermitian symmetric spaces. Let lox (resp. ujy) 
denote the Kdhler form, normalized so that the minimum sectional curvature 
equals — 1 . Say p (or F) is positively maximality preserving if 



sx{X,X') = X-X' 



3. Tightness 



1 



F*ujx 



1 



rank(X) 



rank(y) 
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Say p (or F) is merely maximality preserving if above equality holds up to 
sign. 

Clearly, if this is the homomorphism (f) '.V ^ H is maximal if and 

only a po (j) is. Furthermore, positively maximality preserving maps do not 
change the signs of Toledo invariants. 

Lemma 3.2. Isometric and holomorphic emheddings between equal rank 
Hermitian symmetric spaces are positively maximality preserving. 

Example 3.3. The embedding T>p^p ^ Pp^g between Grassmannians, corre- 
sponding to the embedding SU{p,p) ^ SU{p,q), is isometric and holomor- 
phic, and thus positively maximality preserving. 

Example 3.4. The embedding of symmetric spaces Qn Gn+i, correspond- 
ing to the embedding S0*{2n) ^ 50*(2n+2), is isometric and holomorphic, 
and thus, if n is even, positively maximality preserving. 

Example 3.5. The embedding of symmetric spaces Sn ^ correspond- 
ing to the embedding Sp{2n,'K) ^ Sp{2n-\-2,M), is isometric and holomor- 
phic, and thus never maximality preserving. 

We also need to understand when maximality is preserved under linear 
algebraic operations. Let us start with an easy case. 

Lemma 3.6. Let T be a surface group. Let Iq be a unitary 1-dimensional 
representation ofT and W a sesquilinear representation ofV . Then Hom{Io, W) 
is a maximal representation if and only if W is. 

Proof: Hom{Io,W) and W are isomorphic as projective representations. 
Since U (W) acts on the symmetric space of SU{W) via its quotient PU (W), 
maximality is a projectively invariant property. ■ 
But we shall need a more general case in subsection 7.2. The following 
Lemma is a preparation for Lemma 3.8. 

Lemma 3.7. Let T be a surface group. Let W , W be sesquilinear repre- 
sentations ofT. Then 



Proof: 1. Passing from 14^ to changes the sign of the complex structure 
on the symmetric space X of SU{W). This changes the sign of the Kahler 
form, and thus the sign of Toledo invariants. 

2. Let Y, Y' be the symmetric spaces of SU (W) and SU {W) respectively. 
When y X y is mapped to the symmetric space X of SU{W © W), the 
Kahler form ojx of X restricts on the complex totally geodesic manifold 
y X y to ioy + ojy', so Toledo invariants add up. ■ 

Lemma 3.8. Let T be a surface group. Let V be a unitary representation of 
r and W a sesquilinear representation ofT. Then Hom{V, W) is a maximal 
representation of T if and only if W is. 



(1) T{W) = - 

(2) T{W © W 



T{W). 

') = T{W)+T{W'). 
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Proof: Let : T — )• [/ {HomiV, W)) denote the Horn of the two given rep- 
resentations. Changing the given action on V into the trivial representation 
gives rise to a representation ip : T ^ U{Hom{V,W)). Let us show that 
4> and ijj have equal Toledo invariants. Split W = © W~ into positive 
definite and negative definite subspaces. Then Hom{V,W~^) is a maximal 
positive definite subspace of Hom{V, W), i.e. a point in the Hermitian sym- 
metric space X associated to SU{Hom(V,W)). Its U{V) x U{W)-oihit Y 
is totally geodesic in X, and thus contractible. Therefore, one can choose 
an equivariant map / : S — )■ X whose image is contained in Y. Since 
U(V) fixes Y pointwise, / is equivariant with respect to both (j) and ip. 
Therefore the corresponding Toledo invariants are the same. Since ■0 is a 
direct sum of dim(y) copies of the action on W, Lemma 3.7 applies, so 
T{Hom{V,W)) = dmi{V)T{W). Since rank(X) equals dim(F) times the 
rank of the symmetric space associated to SU{W), ip, and thus (j), is maxi- 
mal if and only if the original T action on is. ■ 

3.2. Tightness. Tightness theory is a way to draw strong consequences 
from the existence of maximal representations. 

Definition 3.9. (Burger, lozzi, Wienhard, [I]). Let G be a reductive Her- 
mitian group, i.e. a connected reductive Lie group in which the center is 
compact and such that the symmetric spaces Xi associated to all simple 
noncompact factors is Hermitian. Normalize the metric on Xi so that the 
minimum holomorphic sectional curvature equals —1. Let Kq denote the 
bounded continuous cohomology class on G defined by integrating the Kdhler 
form of X = WXi on triangles with geodesic sides. Let T he locally compact 
group. Say a continuous homomorphism cp : T ^ G is tight if 



Example 3.10. Maximal homomorphisms of surface groups to reductive 
Hermitian groups are tight. 

3.3. Maximality preserving versus tight. Here is the basic mechanism 
which makes tightness enter our arguments: if i;^> : F — t- G is maximal and 
factors through p : H ^ G, then p is tight. There is a converse statement. 

Proposition 3.11. Let H , G be reductive Hermitian groups. Assume that 
the symmetric space associated to H is irreducible. Let p : H G be a 
continuous homomorphism. Then p is maximality preserving if and only if 
p is tight. 

Proof: This follows from Proposition 2.12 of [4]. ■ 

Example 3.12. (Example 8.7 of [I]). The obvious embedding s SU{n,n) ^ 
Sp{in,R) and SO* (in) Sp{8n,R) are tight. It follows that SO* (An) 
SU{2n, 2n) is tight. All three emheddings are thus maximality preserving. 




K, 



h 
G 



Direct proofs of these facts will be given in an appendix. Lemmas 10.2 
and 10.3. 
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3.4. Consequences of tightness. 

Lemma 3.13. (1) Let H <Z G he connected real algebraic groups. If G 
is reductive Hermitian and the embedding H ^ G is tight, then H 
is reductive Hermitian too. 
(2) Let p : G ^ G' be a tight homomorphism between reductive Hermit- 
ian groups. If the kernel of p is compact and G' is of tube type, so 
is G. 

Proof: This is a combination of Theorems 7.1 and 6.2 of [4]. ■ 

Lemma 3.14. Let V , V' be vectorspaces equipped with nondegenerate sesqui- 
linear forms. Assume that Hom{V,V'), equipped with the natural sesquilin- 
ear form 

(/,/')^Trace(r o/O, 

(here, f* denotes the adjoint with respect to the sesquilinear forms on V 
and V), has vanishing signature. Assume that the induced homomorphism 
U{V) X U{V') U{Hom{V,V')) is tight (see Definition 3.9). Then one of 
V and V' is definite and the other has vanishing signature. 

Proof: Since U{Hom{V,V')) is Hermitian of tube type, Lemma 3.13 im- 
plies that U{y) X U iy') is of tube type, up to compact groups. This implies 
that each of the sesquilinear vectorspaces V and V' is either of vanishing 
signature or definite. Clearly, if both are definite, U{V) x U{V') is compact 
and tightness is out of sight. 

Let us show that V' and V' cannot both have vanishing signature, i.e. 
one of them must be definite. For this, we use the tightness criterion of 
[4], Corollary 8.2. Let Ji denote the generator of the center of the maximal 
compact subgroup of U(y) which defines the complex structure on the sym- 
metric space of U{y). In a splitting of ^ = V~ in a sum of positive 
(resp. negative) definite subspaces. 



Jv 








t T 

2 



Use a similar splitting V' = V'^ © V' and split accordingly Hom{V,V'), 
whose elements become block matrices N = \ ^ | . The action of 



C 

Uv X Uv is {U,U',N) ^ U'-^NU. The linearized action is {U,U',N) ^ 
—U'N + NU. The image of {Jv, Jy) under the linearized action is i— ?> 

/O o\ 




' -f), a map whose matrix is 
zO 



in a splitting of Hom{V, V) 



-i 
\0 ij 

into a positive subspace Hom{V~^ , V'~^)(BHom{V~ , V'^) and a negative sub- 
space Hom(V~^ ,V'^) © Hom{V~ ,V'~^). Therefore the inner product with 
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the element 











\ 


2 




i J 











2 




i T 

2-1 dv 





[ " 





2 




1 J 

2 iy. J 



vanishes. According to [1], CoroHary 8.2, the homomorphism f/(l/)x?7(y) 
U {Hom{V, V')) is not tight. We conclude that one of V and V has vanishing 
signature and the other one is definite. ■ 

The following example of non tight embedding will be useful in the proof 
of Corollary 8.7. 

Lemma 3.15. The injection l : 0(2, 2) ^ U{2,2) is not tight. 

Proof: 0(2, 2) C U (2, 2) is the fixed point set of conjugation a, i.e. aot = 
a. a induces an orientation reversing isometry of the symmetric space X 
of C/(2,2), which changes the sign of the Kahler form, (j*ux = —^x- 
induces a totally geodesic embedding of symmetric spaces, still denoted by 
L -.Y ^ X. Since l*ujx = i*(y*'^x = —l-*^x, '^*'^x = 0, t* is not isometric in 
bounded continuous cohomology, so i is not tight. ■ 



3.5. Alternative definition of balancedness. One may replace symplec- 
tic structures by sesquilinear structures in the definition of balancedness. 

Proposition 3.16. Let G he a semisimple real algebraic group. Let T be a 
surface group. Let (f) : T ^ G be a homomorphism. Let c denote the center 
of the centralizer of <j)(T). Let Zg{c) denote its centralizer in G. Let A be 
a pure imaginary root of the adjoint action of c on g (g) C. Let Qx denote 
the corresponding root space, equipped with a sesquilinear form s\ and its 
imaginary part, the symplectic form Oa . Then the symplectic representation 
r — )• Sp{gx, i^x) is maximal with positive Toledo invariant if and only if 

(1) sx has vanishing signature; 

(2) the sesquilinear representation T — > U{gx,sx) is maximal with posi- 
tive Toledo invariant. 

Lf this is the case, then the homomorphism Zg(c) — )■ U{gx,sx) is tight. 

Proof: 1. Assume that T — t- Sp{gx,^x) is maximal. It factors through 
U{gx,sx)- Thus the inclusion U{gx,sx) ^ Sp{gx,i^x) is tight. Lemma 3.13 
implies that U{gx,sx) has tube type, i.e. the signature of sx vanishes. 

2. According to Example 3.12, the embedding U{gx,sx) ^ Sp{gx,^x) 
is positively maximality preserving, so maximality and positivity of Toledo 
invariant do not change when passing from symplectic to unitary groups. 

3. The maximal representation F — )■ U{gx,sx) factors via the homomor- 
phism Zg{c) — )■ U{gx,sx), which must be tight itself. ■ 
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Now we can explain how balancedness will be analyzed in the sequel. 
For classical simple Lie groups, root spaces 0a turn out to be expressible 
as mapping spaces Hom{V,V'), centralizers Zg{c) are products of classical 
simple Lie groups. Proposition 3.16, combined with Lemmas 3.13 and 3.14, 
restricts the possible groups involved, as will be seen in sections 7 and 8. 

4. Classical simple Lie groups 

4.1. Definition. Classical simple real Lie groups are special linear groups of 
division rings D with center M and special unitary groups of nondegenerate 
binary forms over D, see the appendix in [1 4]. 

There are only 3 such division rings: M, C and H, leading to 3 special 
linear groups, S'L(n,]R), SL{n,C), and 5L(n,IHI). 

Let i be a continuous (anti)-automorphism of D: i can be identity, com- 
plex conjugation or quaternionic conjugation. Let e = ±1. By a (t, e)- 
symmetric binary^ form on a right D-vectorspace V , we mean a D-valued 
M-bilinear map h : V x V D such that 

(1) for all V, v' gV and all q G D, h{v,v'q) = h{v,v')q; 

(2) for all V, v' € V, h{v',v) = €L{h{v,v')). 

Note that from (2), it follows that H{vq,v') = qH{v,v'). Say h is nonde- 
generate if the only v (zV such that h{v, v') = for all v' G V is 0. 

The group U{V,h) consists of right L)-linear self maps of V which preserve 
h. We are interested in the special unitary group SU{V, h) = UiV, h) D 
SL{V,D). 

When D = R, the only choice for l is identity, leading to 

• if e = 1, real special orthogonal groups, indexed by dimension and 
signature and denoted by SO{p, q), where p + q = dim^{V)] 

• if e = — 1, real symplectic groups, indexed by dimension only and 
denoted by 5p(n,M), where n = dim^iV) is even. 

When D = C there are two choice for l. If l is identity, this leads to 
complex special orthogonal (if e = 1) and symplectic (if e = —1) groups, 
indexed by dimension and denoted respectively by SO{n,C) and Sp{n,C). 
If L is complex conjugation, e = ±1 lead to groups called special unitary 
groups, indexed by dimension and signature and denoted by SU{p, q), where 
p + q = dimc(y). 

When D = M there are two choices for l. If z. is identity, no nonzero 
(t, e)-symmetric forms exist. If l is quaternionic conjugation, this leads to 

• if e = 1, quaternionic unitary groups, indexed by dimension and 
signature and denoted by Sp{p, q), p + q = dime(y); 

• if e = — 1, quaternionic skew-unitary groups, indexed by dimension 
only and denoted by SO*{2n), where n = dimiH[(l^) is even. 



To avoid confusion, we keep the words Hermitian for positive definite forms, and 
sesquilinear for the complex case. 
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4.2. Real forms of complex groups. We shall be mainly concerned with 
the 7 families of non complex groups, 3 attached to binary forms on real 
vectorspaces, 1 on complex vectorspaces and 3 on quaternionic vectorspaces. 
Each of these groups is obtained as the fixed point set of an anti-C-linear 
involutive automorphism o" of a complex Lie group, as we now explain. 

Consider first the complex unitary family U{p,q). Here, the data is a 
nondegenerate sesquilinear form s on a complex vectorspace V. For / G 
GL{V), let /* denote the s-adjoint of /, defined by 

yv,v'eV, sifiv),v') = siv,riv')). 

Then 0"(/) = {f*)~^ is an anti-C-linear involutive automorphism of GL(V) 
and of SL{V). The fixed point set of a in SLiV) is SU{V, s). 

The 6 remaining families admit a common construction. 

Given a real vectorspace VJr, let V = VkCSiC and let r =conjugation. Note 
that If 6 is a nondegenerate quadratic or symplectic form on V^, 

let 5 = 6 (g) C. 

Given a right quaternionic vectorspace Vh, pick a basis j, k) of H, use 
right multiplication by i to turn Vm into a complex vectorspace denoted by V. 
Let r be right multiplication by j. Note that = — r. For q = a + jb G H, 
a, 6 € C, denote by C(g) = a. If /i is a nondegenerate ( ~ , e)-symmetric 
binary form on Vh, let, for v, v' € V, 

B{v,v')=C{h{vj,v')). 

Then B is a nondegenerate (-e)-symmetric C-bilinear form on V. 
In both cases, r is anti-C-linear, = tjt for some r] S ±1, and 

B{t{v),t{v')) = B{v,v'). 

For / e EndciV), set 

Then a is an anti-C-linear involutive automorphism of the algebra 01(1^) = 
Endc{V) and of its subgroups SL{V) and 0{V,B). The fixed point set of 
a in SL{V) is SL{Vr) (resp. SLiVm)). The fixed point set of a in 0{V,B), 
i.e., Tof = foT,isU (Vm., b) (resp. U {Vu, h)). This construction yields 6 of 
the 7 families of non complex classical groups. 

4.3. Consequences for roots. The special form of the involution a for 6 
of the 7 families of noncomplex groups has the following consequence. 

Lemma 4.1. Let G belong to one of the above 6 families of classical simple 
Lie groups. Let H C G be a reductive subgroup. Let c C Q be the center of 
its centralizer. Let I be a root of c on C". Let Li denote the corresponding 
root space. Then i is a root and Lg = t{L(). Furthermore, 

(1) ifG = SL{n, R), 0{p, q) or Sp{n, M), L^ = h; 

(2) if G = SL{n,M), SO*{2n) or Sp{p,q), L^ + Lg is a quaternionic 
subspace; 
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(3) If t' ^ {^,^}, h + le '^^^ + ^i' ^'"^ orthogonal with respect to the 
{i^e)- symmetric binary form h, and therefore nondegenerate. 

Proof: Let Z € c and v G Ip. Then 



Ziriv)) = t{Z{v)) = T{i{Z)v) = i{Z)T{v), 

showing that = t{I(). For the 3 real famihes, r is conjugation, thus 
II = li- For the 3 quaternionic families, r is right multiplication by j, 
so li + Ii is stable by right multiplication by i and j, i.e. a quaternionic 
subspace. 

Since h is G-invariant, for all v, v' (^V and Z € c, h{Z{v),v')+h{v, Z{v')) = 
0. If u G li and v' £ Ig', 

= h{Z{v),v') + h{v,Z{v')) 

= h{vi{Z),v') + h{v,v'e'{Z)) 

= h{v,v')e{Z) + h{v,v')£'{Z) 

= h{v,v'){£ + £')iZ). 

This implies that h{v, v') = if f ^ {i, i}. ■ 

4.4. Killing form. Here is a formula for the sesquilinear structure appear- 
ing in Theorem 2.2, valid in all cases. 

Lemma 4.2. Let C s[(n,C) be a complex Lie subalgebra o/s[(n, C). Let 
a be an anti-C-linear involutive automorphism of , with fixed point set q. 
The natural sesquilinear form s on q®C arising from the Killing form of q 
(see section 2) is proportional to 

s{X,X') = TT&ce{a{X)oX'). 

Proof: Since a is anti-C-linear, for / € g, (y{if) = —icr{f) = —if ■ It 
follows that the — 1-eigenspace of a in g*^ is iQ. The map 

Q ^g®L, A ( ^ , — ) 

is an isomorphism. It pulls back conjugation on g C5 C to o" on g*^, thus 
it pulls back the sesquilinear form X • X of g (8) C to cr{X) ■ X on g*^. The 
Killing form on g^ is proportional to the restriction to of the Killing form 
of s[(n, C), whence the formula Trace((T(X) o X). ■ 

4.5. Flexibility of compact and complex Lie groups. For complete- 
ness' sake, we recall here the treatment of compact and complex Lie groups 
from [1 2]. 

Proposition 4.3. Let G be a compact or complex semisimple Lie group. 
Let c C g 6e the center of the centralizer of a reductive subgroup. Then c is 
balanced. 
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Proof: Let X e Q(E}C, X = f + ig. Then s{X, X) = X ■ X = f ■ f + g ■ g. 
If G is compact, the Kihing form is negative definite, so is s. For all roots 
A of c in the adjoint representation, the sesquilinear form s\ on the root 
space 0A is negative definite. No root has vanishing signature. According to 
Proposition 3.16, P is empty so c is balanced. 

If G is complex, centralizers are complex Lie subgroups, c is a complex 
vectorsubspace, and roots A are C-linear maps. None of them is pure imag- 
inary (i.e. takes pure imaginary values on c). Thus P is empty so c is 
balanced. ■ 

5. Complexified centers of centralizers in SL{n,C) 

The first step is to list the possible complexified centers c^C and describe 
the root structure. This depends only on the complexified Lie algebra. 

Lemma 5.1. Let q be a real Lie algebra. Let H C Q be a subset. Let ZZq(H) 
denote the center of its centralizer. Then 

ZZ^{H)®C = ZZg^c{H). 

Our approach consists in using the standard complex representation V 
of eg" C. The root space decomposition of V under c (8) C = ZZg0£{H) 
is obtained from the isotypical decomposition of V under H. This easily 
provides us with the decomposition of End{V) under c (8" C, and then of the 
invariant subspace g (8) C C End{V). 

In this section, we treat the model case of SL(n,C), and in the next 
section, the more elaborate cases 0(n, C) and Sp{n,C). 

5.1. i?- modules. 

Definition 5.2. Let H be a group. The data of a finite dimensional com- 
plex vectorspace and a homomorphism of H onto a reductive real algebraic 
subgroup of Gl{V) is called an i?-module. 

Note that i?-invariant subspaces of i?-modules are again ff-modules. A 
i?-module is irreducible if it has no proper i^-submodules. 

Definition 5.3. Let E{H) denote the set of equivalence classes of H -modules. 
Given an H-module V and vr € E{H), let the n -isotypical component of 
V , denote the sum of all submodules of V belonging to the equivalence class 
vr. 

The following Lemma is well known, but we give a full proof since it serves 
as a model for orthogonal, symplectic and unitary versions of it given in the 
next section. 

Lemma 5.4. Any H-module V splits as a direct sum of its isotypical com- 
ponents 

■KeE{H) 
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Proof: Let W C In he an irreducible invariant subspace. Since H is 
reductive, for each invariant subspace Z of In belonging to the class vr, 
there exists an ff-invariant complement to Z, thus an i?-invariant projector 
pz '■ In ^ Z. Since such submodules Z generate In^ for at least one such Z, 
Pz(W) / 0, thus pz{W) = Z and W belongs to vr. 

More generally, if F C E{H) is a subset, and W C J2neF 
reducible invariant subspace, then W belongs to one of the classes in F. 
Indeed, otherwise pz{W) = for all invariant subspaces Z whose class be- 
longs to F, and these generate '^n€F -^t- particular, for every vr € E{H), 
I-K n Ylin'jtTT -^t' ~ which shows that the sum Ylin&E{H) direct. 

Since H is reductive, the invariant subspace @n&E(H) -^•^ admits an in- 
variant complement, which contains an irreducible subspace. This is a con- 
tradiction unless ®n<^E{H) = V- ■ 

5.2. Centers of centralizers in SL{n,C). 

Lemma 5.5. Let H C SL{n,C) be a reductive subgroup. Let c C s[(n, C) 
be the center of its centralizer. Let L denote the set of nonzero roots of 
c in the standard representation o/s[(n,C), and dg the dimensions of the 
corresponding eigenspaces. Then L has dim(c) + 1 elements, which satisfy 
exactly one linear relation, 

del = 0. 

eel 

Furthermore, the map 

Lx L\ diagonal A, {£, £') ^ £ - i' 

is one to one onto the set A of nonzero roots of c in the adjoint representation 
o/s[(n,C). 

Proof: Since H is reductive, the standard action of H on splits into 
irreducibles. Let us group them into isotypical components Ii is the direct 
sum of kg isomorphic irreducible summands. According to Schur's Lemma, 
the group of iif-automorphisms of the representation Ig is isomorphic to 
GL{keX)- Then 

ZGL{n,C){H) = Y\.^GL{It){H\l^) 

^ \{GL{h,C), 

whose center is (C*)^, acting on C" by multiplication by a different constant 
on each Ig. Pick a basis of C" adapted to the splitting C" = ^gle.- Then 
the center c of Zs^n,c.){H) consists of diagonal matrices diag{ai, . . . ,an) 
whose entries corresponding to basis vectors from the same Ig are equal, 
and which sum up to 0. It follows that the elements of L generate c* and 
satisfy only one linear relation, X^^gl ~ ^- particular, if [i, £'), (m, m') 
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are distinct ordered pairs of distinct elements of L, i — £' — m + m' does not 
vanish identically on c. This shows that the map 

L X L\ diagonal A, {£, i') ^ i - i', 

which is clearly surjective onto the set of nonzero roots of the adjoint action 
on s[(n,C), is injective as well. Furthermore, the root space for i — i' is 
Hom{Ii,Ie,). m 

6. CeNTRALIZERS in ORTHOGONAL, SYMPLECTIC OR UNITARY GROUPS 

In this section, the rootspace decomposition of g under the center of the 
centralizer of a reductive subgroup is given, when g = so(n, C) or 5p(n, C). 
This is a first step in handling real forms of these Lie algebras. With little 
extra effort, one can treat simultaneously the case of g = 5u{p,q). This will 
help treating this particular real form of s[(n,C). 

Let {V, B) be a complex vectorspace equipped with a nondegenerate bi- 
nary form B of one of the following three types, 

(1) symmetric bilinear, 

(2) skew-symmetric bilinear, 

(3) symmetric sesquilinear, 

which we denote by (i, e)-symmetric, e = ±1, i = 1 or ~ (identity or con- 
jugation). Note that the combination (i, e) = ( ~ ,—1) makes perfect sense 
but does not bring anything new, since if i? is a skew-symmetric sesquilin- 
ear form, then iB is symmetric sesquilinear. Let O = 0'''''{V,B) denote 
its automorphism group (note that if B is skew-symmetric bilinear (resp. 
sesquilinear), this is a symplectic (resp. unitary) rather than an orthogonal 
group, whence the notation O''*^). Let -ff C O be a reductive real algebraic 
subgroup. In this section, we describe the center of the centralizer of H in 
O. 

6.1. Bilinear and sesquilinear forms. 

Notation 6.1. Let l be a continuous automorphism of C, i.e. either con- 
jugation or identity. Let e G {—1,1}. Let V he a complex vectorspace. A 
{l, e)-symmetric form on V is a real bilinear form B : V xV ^ C such that 
• for X, X' G C, V, v' G V, B{Xv,X'v') = i{X)X' B{v,v'); 
• B{v',v) = €lB{v,v'). 

In other words, if z. = 1 and e = 1, B is symmetric bilinear. If t = 1 and 
e = — 1, i? is skew-symmetric bilinear. If t = ~ and e = 1, i? is symmetric 
sesquilinear. If i = ~ and e = —1, B is skew-symmetric sesquilinear. We 
shall ignore the fourth case, since if S is a skew-symmetric sesquilinear form, 
then iB is symmetric sesquilinear. 

Notation 6.2. 1. Let l be a continuous automorphism of C, i.e. either 
conjugation or identity. Let V be a complex vectorspace. Then V'' means V 
if L = 1, V if L = ~ . Also, V'''* means the dual vectorspace V* if c = 1, V* 
(i.e. the space of anti-C-linear forms onV) if i= ~ . 
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2. Let B be a (l, e)- symmetric form on V. Let \)tB ■ V ^''* denote the 
C-linear map which maps v (zV to the anti-C-linear functional 

Say that B is nondegenerate if jj^ is an isomorphism. If not, its kernel is 
called the nullspace of B. 

Notation 6.3. Given a linear map L : V ^ V'''* , there is an adjoint map 
L^,T . y _^ yc* defined by 

{L''^{v'),v) = {L{v),v'), 

where {v*,v) denotes the evaluation of a linear or anti-linear form v* on a 
vector V (^V . 

If L is e-symmetric, i.e. if L'''~^ = elL, the formula 

B{v,v') = {L{v'),v) 

defines a (t, e)-symmetric form such that (j^ = L. Therefore B and jl^ are 
equivalent data. 

Example 6.4. Let W be a complex vectorspace. Then the tautological 
isomorphism of V = W x W'''* to V'''* gives rise to a tautological {L,e)- 
symmetric form on V, 

{v, V*) ■ {w, W*) = €l{{v* ,w)) + {w* ,v). 

Every two nondegenerate (1, e)-symmetric forms are isomorphic. On the 
other hand, (t, l)-symmetric forms, i.e. symmetric sesquilinear forms, take 
real values on the diagonal, so sign and signature issues arise: two nondegen- 
erate (i, l)-symmetric forms are isomorphic if and only if they have the same 
signature. For instance, the tautological form of Example 6.4 has vanishing 
signature. 

6.2. Bilinear and sesquilinear i^-modules. 

Definition 6.5. Let H be a group. The data of a finite dimensional complex 
vectorspace equipped with a bilinear (either symmetric or skew-symmetric) or 
sesquilinear form and a homomorphism of H onto a reductive real algebraic 
subgroup of its automorphism group will be called a (i, e)-linear i^-module. 

Lemma 6.6. Let W be an irreducible H -module. The space of H -invariant 
bilinear (resp. sesquilinear) forms on W has dimension at most 1. A 
non-zero H -invariant bilinear (resp. sesquilinear) form is automatically 
non- degenerate, and in the bilinear case, it is either symmetric or skew- 
symmetric. 

Proof: Let h be an /^-invariant bilinear form on W . Its nullspace is 
i?- invariant. Therefore 5 is either zero or non-degenerate. Assume b is 
nonzero and denote hy % : W ^ W'''* the corresponding isomorphism (in 
the sesquilinear case, i = conjugation, W'''* = W*). Let b' be an other 
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i?- invariant bilinear form on W. Then L = (ttb)~^ ° ttfe' is an ff-equivariant 
endomorphism of W, thus L is a multiple of identity (Schur's Lemma). This 
shows that b' is a multiple of b. b has a symmetric and a skew-symmetric 
component. They have to be linearly dependant, in the bilinear case (z- = 1), 
this implies that one of them vanishes. Therefore b is either symmetric or 
skew-symmetric. ■ 

Definition 6.7. Say an H-module is (t, e)-orthogonal if it admits an in- 
variant non- degenerate [i^e)- symmetric form. 

Corollary 6.8. The classification of irreducible [l, e) -linear H -modules can 
be deduced from the classification of irreducible H -modules: the forgetful map 
E^'''^\H) — >■ E{H) is onto, the fiber of an equivalence class of irreducible H- 
modules contains 1 or 2 elements depending wether it is {L,e)- orthogonal or 
not. 

Lemma 6.9. Let {V,B) be a (l, e)-linear H-module. Let W and W' be 

distinct irreducible H -invariant subspaces. Assume that W is not orthogonal 
to W . Then W' is isomorphic, as an H-module, to the (conjugate-) dual W'''* 
ofW (in the sesquilinear case, W'''* = W*). 

Proof: The map v i-^ i'^Biv))\w^ ~^ W'''* , is //-equivariant. According 
to Schur's Lemma, such a map is either zero or an isomorphism, and all 
such maps are proportional. By assumption, it does not vanish, thus W 
and W"'* are isomorphic ff-modules. ■ 

Definition 6.10. Let H be a group, V an H-module. Say V is bi-isotypical 
if their exists an irreducible H-module Z such that every irreducible invariant 
subspace W <ZV is isomorphic either to Z or to Z'''* . 

Corollary 6.11. Let iV^B) be a non- degenerate {L,e)-linear H-module. 
Then V canonically splits as an orthogonal direct sum of its bi-isotypical 
components, 

TT 

where, given an equivalence class vr of irreducible H-modules, In,iT''* = 
L^ + Ijri.* is the sum of all irreducible invariant subspaces of V isomor- 
phic either to vr or to vr*^'*. Furthermore, /jr,?!'.* is non- degenerate, and the 
centralizer ZQfy ^-^ {H) of H in the automorphism group is isomorphic to a 
direct product, 

Zo{V,B){H) = JJZo(/^^,„. 

6.3. Examples of bi-isotypical bilinear /sesquilinear i7- modules. From 
now on, we analyze non-degenerate bi-isotypical (i, e)-linear if-modules. 
Here are two examples. 
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Example 6.12. Let Z = vr®'' be an isotypical H -module. Set V = Z x Z''*, 
equip it with the canonical (l, e)- symmetric form 

{v,v*) ■ {w,w*) = ei{{v*,w)) + {w*,v). 

Then the centralizer of H in 0^^'^\V,-) is isomorphic to an orthogonal, 
symplectic or unitary group, if tt and vr^'* are equivalent 

{0{2rX) ^fi=l, e = l, 
:= lsp{2r,C) ifi = l, e = -1, 
[U{r,r) if L= " , e = 1, 

otherwise to a general linear group, 

^o(-)(y,)(^) = G'/(r,C). 

Proof: Pick a basis of W and take r copies of it to form a basis of 
Z. Take the image of this basis under {|s to get a basis of Z*^'*, i.e. for 
e = 1, choose B{ei, Cj) = 1 and e* = tt_B(ei), and then take {ei, • • • ; e^, • • • } 
as a basis for V . For e = —1, choose ei, • • • , e2k so that B{ei, ej+fc) = 1 = 
— i?(ej+fc,ej) and B{ei,ej) = otherwise. Choose a basis for V in this case 
as {ei,- • • ,efc,efc+i,- • • ,e2fc,e^_^^, • • • ,e^^,e^,- • • ,e^}. This gives a basis of 

V in which the matrix of the bihnear/sesquihnear form B equals 

(blocks have size rd where d = dimvr). In this basis, the matrix of an 
element g of H splits into blocks of size d, with the first r diagonal blocks 
equal to TT{g) and the last r equal to 7:{g~^)'''~^ , all other blocks vanish. 

If TT and tt'''* are equivalent, endomorphisms of V which commute with H 
have matrices whose blocks of size d are scalar, i.e. proportional to the unit 
dxd matrix. In other words, they can be written A® 1 where A € Gl{2r, C). 
Such a matrix preserves B if and only if 

i.e. iff A belongs to 0(2r, C) if e = 1, to Sp{2r, C) if e = -1 and to U{r, r) 
if L =conjugation. In other words, 

^o(.0(y,)(^) = 0^'''^(2r,C). 

If vr and tt'''* are not equivalent, endomorphisms of V which commute with 
H preserve the splitting F = © I^l.* and have matrices whose blocks of 
size d are scalar, i.e. proportional to the unit dxd matrix. In other words, 
they can be written (^4 © A') (8) 1 where A, A' G G/(r, C). Such a matrix 
preserves B if and only if A' = {A'''~^)'^. In other words. 



ZoMiV,-)iH) = Glir,C). 
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Example 6.13. Let V = vr®'^' be an isotypical H-module such that vr is {l, e)- 
orthogonal, i.e. preserves a non- degenerate (l, e)- symmetric form h. Let D 
he a real diagonal invertible r x r matrix. Set B = b D, i.e. {V,B) is an 
orthogonal direct sum of r real multiples of the same non- degenerate {i,e)- 
linear H-module. Then the centralizer of H in 0''(y,b^D) is isomorphic 
to an orthogonal/unitary group, 

^0'-^(v,b<g)D){H) = 0''{C^,D). 

Note that in the bihnear case, ©(C", D) = 0{r, C) is a genuine orthogonal 
group, even when e = — 1, i.e. when we deal with skew-symmetric forms. In 
the sesquilinear case, 0^(C^', D) = U{p, q) where p — q = sign(Z)). 

Proof: Repeat the same basis of vr to get a basis of V. The matrix of 
B = b D is diagonal in blocks of size d = dimvr, with diagonal blocks 
equal to real multiples of b. Element g & H acts by a diagonal matrix in 
d X d-blocks, with diagonal blocks equal to iT{g). The centralizer of H in 
Gl(y) consists of matrices with scalar d x d-blocks, i.e. of the form A 1 
for A £ Gl{r,C). (t, e)-orthogonal matrices satisfy 

{A (g) lY'^ib (gD){A^l) = b(g (A^'^DA) = b(gD, 

i.e. A''^DA = D. Thus 

Zo^(,v,B){H)?^0\C\D). 

■ 

6.4. Classification of bi-isotypical bi/sesquilinear ff- modules. There 
are 3 cases, depending wether vr is (i, e)-orthogonal, (t, — e)-orthogonal, or 
neither. In each case, we will need the following lemma. 

Lemma 6.14. Let V = W ® W be a {L,e)-linear H-module where W , W 
are isotropic, isotypic, and the map L : W' — >■ W^'* , L{v') = {^b{v'))\w ^■s 
an isomorphism. Then V is isomorphic to Example 6.12. 

Proof: The iif-map {w + w') i— > {w,L{w')) is an isometry W W — > 
W X W-'*, since 

{v, L{v')) ■ (w, L{w')) = eLi{Liv'),w)) + {L{w'),v) 

= ei((ttsK), + {U{w'),v) 
= €l{B{w,v')) + B{v,w') 
= B{v',w) + B{v,w') 
= B{v + v' ,w + w'). 

■ 

Proposition 6.15. Let {y,B) be a non- degenerate bi-isotypical bilinear H- 
module. Assume n and tt'''* are not isomorphic. Then {V, B) is isomorphic 
to Example 6.12. 
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Proof: Irreducible invariant subspaces of V belong either to tt or vr^'*, which 
do not admit non-degenerate invariant bilinear /sesquilinear forms, thus all 
are isotropic. Lemma 6.9 implies that every two distinct irreducible invariant 
subspaces of It^ are orthogonal. Thus 1-,^ is isotropic, and so is /tt'^* • Since 
V is non-degenerate, : — )■ V"'* induces isomorphisms L : ^ [I^l.*Y'* 
and L' : Ij^i..* {IttY'* related by L"*" = eL' . According to Lemma 6.14, this 
shows that V is isomorphic, as a (t, e)-linear //-module, to Example 6.12. ■ 

Proposition 6.16. Let V = I-^ be an isotypical H-module equipped with a 
nondegenerate e-symmetric bilinear form B. Assume tt is {—e)-orthogonal. 
Then {V,B) is isomorphic to Example 6.12. 

Proof: Since vr is not e-orthogonal, all irreducible i/-submodules of V are 
isotropic. Let W be one of them. A ff-invariant complement to contains 
an irreducible //-submodule W\ which is not orthogonal to W . The i?-map 
L : W' — > W*, v' i-> (tt_B(?^'))|VF is non-zero, thus an isomorphism, and 
Lemma 6.14 implies that W © W' is isomorphic to Example 6.12, in partic- 
ular, it is non-degenerate. Its orthogonal is again non-degenerate, isotypic, 
modelled on a (— e)-orthogonal ff-module. By induction on dimension, V 
is an orthogonal direct sum of copies of Example 6.12, thus isomorphic to 
Example 6.12. ■ 

Proposition 6.17. Let V = I-,^ be an isotypical H-module equipped with a 
nondegenerate {l, e)-symmetric bilinear form B. Assume vr is {l, e)- orthogonal. 
Then iV^B) is isomorphic to Example 6.13. 

Proof: Let us show that V contains at least one non-degenerate irreducible 
invariant subspace. Pick an irreducible iJ-submodule W of V . If it is 
non-degenerate, we are done. Otherwise, W is isotropic. An //-invariant 
complement to W'^ contains an irreducible i?-submodule W' , which is not 
orthogonal to W . If W is non-degenerate, we are done. Otherwise, W 
is isotropic too. Then W © W' is isomorphic io W x W'''* equipped with 
the canonical (i, e)-symmetric form. Indeed, the H-map L : W' — )■ W'''*, 
v' I— > i'iB{v'))\w is non-zero, thus an isomorphism, and Lemma 6.14 applies. 
By assumption, there exists an e-symmetric //-isomorphism M : W ^ W"'*. 
Then the graph Z = {{w, M{w)) \ w € W} of M is non-degenerate. Indeed, 
for V, w £ W, 

{v, M{v)) ■ {w, M{w)) = eL{{M{v),w)) + {M{w),v) 

= 2e{{M{v),w)) 

cannot vanish for all w, unless v = 0. Z is a non-degenerate irreducible 
(i, e)-linear //-submodule of W x W"'*, which embeds isometrically into V, 
this is the required subspace. 

The proof of the proposition is concluded by induction on dimension. 
If V is irreducible, we are done. Otherwise, we just showed that V has at 
least one non-degenerate irreducible submodule, say W. Then the induction 
hypothesis applies to its orthogonal W'^. ■ 
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Corollary 6.18. Let iV^B) he a nondegenerate bi-isotypical {l, e) -linear H- 
module. Then (V,B) is isomorphic either to Example 6.12 or to Example 
6.13. The centralizer of H in the automorphism group of{V,B) is isomor- 
phic to a general linear group in the former case, an orthogonal group in the 
latter. 

6.5. Centers of centralizers in unitary groups. In this case, we directly 
get information on a real form. 

Proposition 6.19. Let (y,B) be a non- degenerate symmetric sesquilinear 
H -module. Let c denote the center of the centralizer of H in U{V,B). Then 
root spaces of V under c correspond to isotypical components /•„- under H . 
They fall into bi-isotypical components BL^^ which are pairwise orthogonal. 
If Itt = BIt^, then the corresponding root of c is pure imaginary. If I-^ 7^ -B/tt, 
then BIj^ = I^. , the corresponding roots i-j^ and i-,^* are opposite. The 
linear relations between roots are generated by l-^ + 1^* =0, vr 7^ vf*. 

Let N be a set which contains exactly one element of each pair {7r,7f*)} 
of equivalence classes of irreducible sesquilinear H -modules occurring in V , 
such that TT 7^ vf*. Then the map (^7r)7reLiv • ^ ~^ C^'^ is onto. 

Proof: If = -B/tt, then, as a sesquilinear i7- module, BI-,^ is isomorphic 
to Example 6.13 and contributes a unitary factor to the centralizer of 
whose center is a pure imaginary subgroup of SL{n,C). It acts on BIj^ by 
multiplication by a pure imaginary number. Thus BIj^ is a root space for 
a pure imaginary root. Otherwise, BIt^ is isomorphic to Example 6.12 and 
contributes a general linear group factor to the centralizer. Its center is a 
complex subgroup of SL{n, C), it acts on /^r (resp. Is-*) by multiplication by 
an unrestricted complex number (resp. the opposite number). This produces 
a subspace c' of c which admits a complex structure, the corresponding roots 
are C-linear and half of them (to avoid the relations + ijf* = 0) provide 
complex coordinates on c'. ■ 

Corollary 6.20. Let L be the set of roots, Lj the subset of roots which take 
only pure imaginary values. Then for every £ G L \ Lj, —(.is again a root. 
Let Ln be a set which contains exactly one element of each pair {i, —i}, 
ieL\Li. Then 

• li and li' are orthogonal unless i' = —i; 

• if i £ Lj, I^ is nondegenerate; 

• the map {()e^LN ■ ^ ~^ is onto. 

6.6. Centers of centralizers in e-orthogonal groups. We continue our 
convention that 




Note that in the latter case, n has to be even. 
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Proposition 6.21. Let {y,B) he a non- degenerate bilinear H -module. Let 
c denote the center of the centralizer of H in 0''{V,B). Under c, V splits 
into root spaces as follows. Each isotypical component I-,^ where vr is an 
irreducible II -module which is not equivalent to its contragredient tt* is a root 
space for a non-zero root i^^. It is isotropic. The sum of all other isotypical 
components constitutes the root space. Bi-isotypical components It^ + /^r* 
are pairwise orthogonal. Relations among non-zero roots are generated by 
the following 

Thus the number of non zero roots is 2dim c. Let L be a set containing 
exactly one element of each pair (7r,7r*). Then (^jfjirGL : c is a linear 

bijection. 

Proof: Let L' be the set of equivalence classes of irreducible ff-modules 
which are isomorphic to their contragredient. Then 

Zo^iy,B){H) = n GL{r^X) X J] 0^(2^--.^), 

ttGL ttGL' 

thus 

ZZo'^{V,B){H) = 0C(id/^ - id/^J, 

i.e. (^7r)7reL : c — ?• is a linear bijection. Furthermore, if vr G L, the 
bi-isotypical component © I-,^* splits into two root spaces relative to roots 
-^Tf and -^Tf* -— -^Ti". B 

Corollary 6.22. Let L denote the set of roots of c on C" . The roots of c in 
its adjoint action on 5o''(y,B) are exactly all differences l — for i, (! € L, 
including 2i if e = —1 or dim(/£) > 1, excluding 2i if li is 1-dimensional 
and e = 1. 

In other words, if L is a set of representatives of pairs of nonzero 

roots, one finds 0, all sums ±1 it £' for distinct £, £' S L, sometimes ±21 
(depending on e and dim(/£)j, and, if is also a root of c on C", all ±i, 
e€L. 

Proof: The Lie algebra 5o{V,B) is the space of B- skew- symmetric endo- 
morphisms of V, i.e. C-linear maps f : V V satisfying, for all v, v' € y, 

B{f{v)y)+B{vJ{v')) = Q. 

The roots of c in the adjoint representation are differences \ = £ — i' 
of roots of c in C". If £' ^ ±£, the root space relative to £ — £' is the 
subspace of i?-skew-symmetric elements of IIom(l£, Ii/) © IIom(I^£i , I^i). 
It does never vanish. Indeed, for every / € IIom(Ii, Ig/), there is a unique 
g € IIom{I-£i , I-e) such that {f,g) is i?-skew-symmetric. The formula for 
9 is 



22 



INKANG KIM AND PIERRE PANSU 



Here is an alternative description of 5o{V,B): mapping / € End{V) to 
the bilinear form b{v,v') = B{f{v),v') identifies EndiV) with the space 
V* ® V* of C-bilinear forms on V and 5o(V, B) with the subspace K'^V* of 
(-e)-symmetric C-bilinear forms on V . The adjoint action of Z G 5o{V) is 

Zh{v,v') = B{[ZJ]{v),v') = -b{v,Zv') -b{Zv,v'). 

If f € Ii, v' € l£i and Z € c, then 

Zb{v,v') = -{i + e'){Z)b{v,v'). 

Therefore the root space relative to 2i identifies with A^I^. It vanishes if 
and only if e = 1 and dim(/£) = 1. ■ 

7. Real forms of SL{n, C) 

7.1. Flexibility in SL{n,R) and SL{n,M). 

Proposition 7.1. Let c C s[(n,]R) (resp. 5l{n/2,M.)) be the center of the 
centralizer of a reductive subgroup of SL{n,M.) (resp. SL{n/2,M)). Let A 
be a pure imaginary root of c in its adjoint action on s[(n,M). Then the 
signature of the Killing form restricted to 3a,r does not vanish. It follows 
that c is balanced. 

Proof: Let C" denote the standard representation of sl(n, C). As in subsec- 
tion 4.2, let t{v) = -u in the complex case, and t{v) = vj in the quaternionic 
case (here, C" = H"'/^ is viewed as a right quaternionic vectorspace). Let, 
for / e s[(n,C), cr(/) = r o / o r^^ Then Fix{a) = Q = 5[(n,M) (resp. 
5[(n/2,EI)). 

Under c, C" splits into root spaces C" = dim{Ii) = di. Roots are 

either real or come in pairs (Lemma 4.1). According to Lemma 5.5, 

every nonzero root A of c in its adjoint action on 5l{n, C) can be uniquely 
written in the form £ — i'. Such a root is pure imaginary if and only if i' = i, 
i.e. X = i — £. The corresponding root space is 

Qx = Hom{I^,Ij). 

Let / S Hom(Ii, Ii), cr{f) = f € Hom(I^, Ii). Pick a basis of Ii and take its 
image by r as a basis of I^. Let M denote the matrix of / in the chosen basis 
ofC". Then the matrix of cr(/) is cr(M), and Trace(o-(/)o/) = Trace(MM). 
Write M = S + A where S is symmetric and A is skew-symmetric. Then 

TVace(MM) = Trace{SS) + Trace(iA) = Trace(5*5) - Trace(AM) 

has signature dim({5}) — dim({A}) = '^^^'^^^^^ _ ^t(d-^-^) _ which is 
nonzero. ■ 

7.2. Flexibility in SU{p,q). 

Proposition 7.2. Let T be a surface group, let (j) -.V ^ SU{p, q) a reductive 
homorphism, let c be the center of the centralizer of <j){T). Assume that c is 
not balanced with respect to (j). Then, up to conjugacy, 4>{T) is contained in 
S{U{p,p) X U{q — p)), and cp is maximal. 
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Proof: Under c, the standard representation of 5l{n, C) splits into root 
spaces, C" = ^ile- The roots of c in the adjoint representation are differ- 
ences £ — £' , and all of them indeed occur. Following Corollary 6.20, split 
the set of roots as L = L/UL^vU— L^v, where Lj is the subset of pure 
imaginary roots. If £ € L^r, 21 is a root in the adjoint representation, and 
it is not pure imaginary. If £ € and i' G Lj^, i — i' is not pure imaginary. 
If £ and £' G L^r, £ — £' is not pure imaginary either, since it factors through 
a surjective map c C^^ and a C-linear form C^^ ^ C. So if L^r is non 
empty, non pure imaginary roots span c*, and c is balanced, contradiction. 
So Ltv is empty, all roots on C"" are pure imaginary, and the corresponding 
root spaces li are nondegenerate and pairwise orthogonal. In the sequel, 
we shall replace roots by there imaginary parts without expressing it in the 
notation. 

Each root A in the adjoint representation can be expressed in a unique 
way as X = i — £', and 

Qx = Hom{h,Ii,). 

A calculation shows that the signature of the natural Hermitian form on 
Ho'm(l£, Iir) equals — sign(/^)sign(/£/). This time, the signature is not auto- 
matically nonzero. So different arguments, based on [4], are needed. 

Assume that the sesquilinear action of T on qx is maximal. The situation 
we are considering is as follows: since H = (l)(T) preserves Ii and preserves 
a sesquilinear form on it, </)(r) C U{Ii), and similarly for Ig', hence <^(r) — > 
U{Ii) X U{Ii') and we obtain 

^{T) c Zg(c)^^-^^ X U{Ie) 

U{qx) = U{Homih, le)) = U{h ® It,) 

where / is tight. According to Proposition 3.16, the sesquilinear space 
Hom{l£, III) must have vanishing signature. Also, the homomorphism Zg{c) — >■ 
U{Hom{Ig, III)) must be tight. Lemma 3.14 implies that one of Ii and Iii 
has vanishing signature and the other one is definite. Say Ii is definite, 
for instance. According to Lemma 3.8, maximality of Hom{Ii, Iii) implies 
maximality of the T action on J^/, with Tx = dim{Ii)T{Iii). If instead Iii is 
definite, Tx = -T{Ii)dim{Iii). 

Let D (resp. E, resp. O) denote the set of roots £ such that Ig is definite 
(resp. has vanishing signature, resp. has non vanishing signature). As 
in definition 2.1, let P denote the set of roots A such that px is maximal 
with positive Toledo invariant. Equivalently, of differences £ — £',£ ^ D, 
£' € E, such that Iii is maximal with positive Toledo invariant. Let be 
the complement of ibP. 

Let us show that if O is non empty, c is balanced with respect to (p. 
Indeed, let £q £ O. Then for all £' ^ £o, for X = £q — £' , px is not maximal, 
thus £o — £' G N. Since the roots £ span c* (giC and satisfy the extra equation 
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J2e dii = 0, spanc({£o io}) = c* O C. Since spanc(iV) = c* ® C, c 

is balanced with respect to (p. 

Prom now on, we assume that O is empty. If D or E is empty, there 
is no room for pairs for £ D and i' E, so P is empty, hence 

balanced. Therefore we assume that both D and E are non empty. Let 
(resp. Le) denote the span of all differences i — i' for I ^ D and i' ^ D 
(resp. for £ ^ E and I' G E). Then d\m{L£,) = card(L') — 1, dim(L£;) = 
card(£') — 1 by Lemma 5.5. Since L^) C spanc(Z)), C span£(ii^) and 
spanc(I?) n spanc(£') is the line generated by ^i^f)di£, Ld n Le = {0}, 
thus dim(L£) + Le) = card(L') + card(£') — 2 = dim(c) — 1. In the quotient 
space c* (Si C/ Ld + Le, all elements of D (resp. of E) are mapped to the 
same vector if) (resp. £e), and £d ^E- Again, if one of the £ — £',£€ D, 
£' & E belongs to N, c* (8) C/N vanishes, so c is balanced with respect to (p, 
contradiction. 

Therefore, all £ - £', i e D, £' e E, belong to ±P. If there exists two 
pairs {£,£'), £ G D, £' G E, such that Qm{£ — £') have opposite signs in 
c* /''^m{LE) + Le), then c is balanced with respect to (/), contradiction. 

Otherwise, all £ — £',£ ^ D, £' £ E, belong to ±P and those which 
belong to +P project to c* /Qm{LD + Le) with equal signs. This implies 
that the direct sum representation ^j^^e maximal. In other words, 
0(r) C U{p,p) X U{q — p) is maximal. The symmetric spaces T^p^p and 
"Dp^q corresponding to SU{p,p) and G = SU{p,q) have equal ranks and the 
embedding T>p^p ^ T^p,q is isometric and holomorphic. Therefore Example 
3.3 implies that, viewed as a homomorphism T — > G, is maximal as well. 



7.3. Rigidity in SU{p,q). The centralizer of SU{p,p) in SU{p,q), q > p 
is U{q — p) with center c = u(l) generated by Z = 



p+cj 

There is only one nonzero pair of roots =bi, giving rise to the root space 
i7omc(C'^~^, C^^). The sesquilinear form Sj on H omc{C'^~^ , C^^) is, in an 
S'C/(p,p)-invariant manner, the direct sum of q — p copies of the U{p,p)- 
invariant Hermitian form on C^^. Therefore the corresponding Toledo in- 
variant is equal to q — p times the Toledo invariant obtained for q = p + 1. 
In this case, the centralizer of c is U{1) x SU{p,p), acting on the real root 
space via the standard complex representation of its second factor. The 
representation pi induced on a surface subgroup L C C/(l) x SU{p,p) is 
maximal if and only if the projection of T to SU{p,p) is maximal. Thus a 
maximal surface subgroup of SU{p,p) is not flexible in SU{p,q), q > p, as 
is well known. Such subgroups exist (Theorem 1.8) and are known to be 
automatically discrete, [1], [3]. 



8. Real forms of 0(n, C) and Sp{7i,C) 



8.1. Non pure imaginary roots. 
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Lemma 8.1. Let G be a real form of 0''{n,C). Let H d G he a reductive 
subgroup. Let c C g 6e the center of its centralizer. If one of the roots of c 
on C" is not pure imaginary, then c is balanced. 

Proof: By contradiction. Assume that c is not balanced and at least 
one of the roots of c on is not pure imaginary. Consider the torus 
C(8)C C 50*^(71, C) and the set LU—L of its nonzero roots on C" (Proposition 
6.21). Write L = Lj U Ljy where Lj C L denotes the set of roots whose 
restriction to c takes only pure imaginary values and L^r its complement. 
By assumption, L^v is nonempty. 

Assume first that Lj is nonempty too. None of the roots A = ±1 it i' , 
i £ Lj, i' G Ltv, is pure imaginary, so none of them belongs to ibP. They 
generate span(L) = c*, thus c is balanced, contradiction. Therefore, L = Ln. 

If every root of adjoint action is non pure imaginary, it is balanced. Hence 
they do not span c*, which implies that there exists f G c such that X{v) = 
for all roots A of c in the adjoint representation which are not pure imaginary. 
For every distinct i and i' (z L, one of ^ — / and £ + i' is not pure imaginary, 
thus i'{v) = ±£{v). 

Assume that there exists i £ L such that £ or 2i is a root of an adjoint 
action, then i{v) = as well, which implies that i'{v) = for all £' € L, and 
V = 0. In other words, in that case, non pure imaginary roots span c*, and 
c is balanced, contradiction. Therefore, 

• is not a root of c on C" (i.e., all root spaces correspond to elements 
of ±L); 

• for every root £ £ L,2£ is not a root of c in the adjoint representation. 

This implies that e = 1 and all root spaces le have dimension 1, i.e. c <8> C 
is a maximal torus of so(n,C). Its centralizer in so(n,C) is abelian, so are 
Zg{c) and H, up to finite index. A homomorphism from an abelian group 
cannot be tight, so no symplectic action on root spaces can be maximal, P 
is empty, and c is balanced again, contradiction. ■ 

8.2. The sesquilinear structure. 

Proposition 8.2. Let G be a real form of ©"^(n, C). Let H C G be a 
reductive subgroup. Let c C Q be the center of its centralizer. 

(1) Let £ he a pure imaginary root of t on C". Let Ii denote the corre- 
sponding root space. Then Ii inherits a ZG{c)-invariant nondegen- 
erate sesquilinear form sg. 

(2) Assume that all roots of c on C" are pure imaginary. Then the 
centralizer Zg{c) of c in G is a product of the fixator of the orthogonal 
of the root space Iq and of unitary groups, 

Zg{c)^Gj^ X J] [/(/,, s,). 

Proof: 1. Let si{v,v') = B{t{v),v'). This is an rye-symmetric sesquilinear 
form. It is nondegenerate on Ii because B is and all root spaces but /_£ = 



26 



INKANG KIM AND PIERRE PANSU 



t(/^) are i?-orthogonal to Ii. If 5 G Zg{c), i.e g belongs to G and commutes 
with c, then g leaves all root spaces I±e invariant, it commutes with r, and 
is isometric for B, thus it is isometric for S£. 

2. Let g & G commute with c. Let go € GL{n, C) be the element which 
coincides with g on Iq and fixes the sum of all Ii, £ G itL, i.e. Iq. Then 
go G G. Indeed, go commutes with r and preserves B. Also, restrict g to 
each l£, i £ L. This yields an injective homomorphism Zg{c) — s- Gj± x 
UeeLUiIi,si). 

Conversely, let go G he an element which fixes all le- For each i £ L, 
pick an element gi E U{l£,se), extend it to I-i so that ge = t o gi o r"^, 
extend this map trivially to other Ig/ and to Jq. The obtained linear map 
belongs to Zg{c). Indeed, it preserves B, commutes with r and preserves 
each Ig. Multiplying go with the ggs yields the inverse isomorphism Gj± x 

Lemma 8.3. Let G be a real form of 0^{n,C) . Let H d G he a reductive 
subgroup. Let c C Q be the center of its centralizer. Let i, i' be distinct 
pure imaginary roots of c on C"", and X = i — i' the corresponding pure 
imaginary root of c in its adjoint action on g. Let Ig, Igi and Q\ denote the 
corresponding root spaces. 

1. If i' ^ —I, then, as a sesquilinear ZG{c)-module, Qx is isomorphic to 
Hom{Ie, Igi) equipped with the natural sesquilinear form 

(/,/')^Trace(r o/'), 

where f* € Hom^Igi , Ig) is the adjoint of f with respect to the sesquilinear 
forms se and sgi . 

2. If i' = —i, then, as a sesquilinear ZG{c)-module, Q\ is isomorphic to 
the subspace of —e- symmetric forms in the space I^<^I^ of C-bilinear forms 
on Ig, equipped with its natural sesquilinear form 

(6, h') ^ TYace((tt.J-i o (y ^ o {%^)-^ o y). 

Proof: 1. Assume first that i and / are linearly independant. Accord- 
ing to Corollary 6.22, is the space of i?-skew-symmetric elements of 
Hom{l£, Iii)(BHom{I_£i , I^i). Given / G Hom{l£, Ii') andg S IIom{I_£i , I^i), 
^ = if yd) is i?-skew-symmetric means that for all v £ Ig and w G I-e, 

B{fiv),w) + B{v,g{w))=0. 

Since fts identifies I-£ with the dual of Ii, given / € Hom{Ii, Igi), there exists 
a unique g G Hom(I^£i , I^i) such that X = {f,g) is i?-skew-symmetric. 
This shows that, as a ZG'(c)-module, Qi-i' is isomorphic to Hom{l£, Ig'). 
Since (7{X) = [t o g o t~^,t 0/0 t~^), 

a{X) o X = r]{T o goT o f,T o f OT o g), 

therefore 

Trace((T(X) o X) = r]Trace{T o g o t o f) + r/Trace(T o f o t o g) 
= 2r/Trace(r o o r o /). 
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Uv e li, 

Si{v,T o g OT o f{v)) = B{t{v),t o g o T o f{v)) 
= B{v,goTo f{v)) 
= -Bifiv),Tof{v)) 
= -eB{Tof{v)J{v)) 
= -e s,{f{v)J{v)) 
= -esi{v,f*of{v)). 
Summing over an orthogonal basis for si yields 

Trace(r o g o t o f) = — eTrace(/* o /), 

hence the Killing sesquilinear form on q\ is proportional to Trace(/* o /). 

2. Let / G Hom{l£,I-i). Then / belongs to so(y,B) (/ is _B-skew- 
symmetric) if and only if /* = —a{f). Indeed, 

yv,wele, B{f{v),w) + B{v,f{w))=0 

<^ W e le, S-i{f{v),T{w)) + Si{v,T O f{w)) = 

^ yveIi,WeI.i, S_iifiv),v')+Si{v,TofoT'\v'))=0 

^ f* = -T0f0T-\ 

Thus, for i?-skew-symmetric /, 

Trace((T(/) o /) = Trace(r o / o o /) = — Trace(/* o /). 

Now / is S-skew-symmetric if and only if the bilinear form b{v, v') = 
B{f{v),v') on li is (-e)-symmetric. Since ftfi = ° ttfi = vf^ o o (j^^ 
and r = {U~'°(W°K, 

■ 

Lemma 8.4. Let V be an n-dimensional complex vectorspace equipped with 
a nondegenerate sesquilinear form of signature s. Then the signatures of 
the induced sesquilinear forms on V* ^V* , S^V* and A'^V* are equal to s"^ , 
(s^ + n)/2 and (s^ — n)/2 respectively. 

Proof: Fix a basis of y . li D denotes the matrix of the sesquilinear form S 
in this basis, and 6, b' the matrices of two bilinear forms on y, the induced 
sesquilinear form on V* ® V* is 

Trace{V D-^b'D-^). 

One can assume that S is diagonal with entries dm = 1 (p times) and —l{q 
times), n = p + q, s = p — q. Then 

m,m' 
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is diagonal again. The signature of the whole space is ^/ dmdm' = 

s^. The signature of the subspace A'^V* of skew-symmetric forms is 



8.3. Unbalanced centers of centralizers. Recall that, when c C g is 
the center of the centralizer of a homomorphism of a surface group to G, 
P denotes the set of roots A in the adjoint representation such that the 
sesquilinear action on the root space 0a is maximal with positive Toledo 
invariant. 

Lemma 8.5. Let G be a real form of 0''{n,C). Let (p -.T ^ G be a reductive 
homomorphism. Let c C Q be the center of its centralizer. Assume all the 
roots are pure imaginary. Let i, i' be distinct nonzero pure imaginary roots 
of c on C", and X = i — i' the corresponding pure imaginary root of c in its 
adjoint action on q. 

(1) Lf i' ^ —i, and A G itP, then one of sg and sg' has vanishing signa- 
ture and the other is definite. 

(2) Lf i' = —i, then X = 2£ either is not a root or does not belong to ibP. 

Proof: If A G P, 0a has vanishing signature. So does U{L[om{Li, Lg')) 
(resp. U{A^{Li))) with its natural sesquilinear form, according to Lemma 
8.3. In particular, these groups are of tube Hermitian type. Furthermore, 
the Zg{c) action on 0a is tight. According to Proposition 8.2, Zg{c) is a 
product of groups. 

1. If i' 7^ —i, among the factors, only U{Li,Si) and U{Li',Si') act non 
trivially on 0a, thus the morphism 



must be tight. Lemma 3.14 applies and one of the left hand groups is 
compact and the other has vanishing signature. 

2. If i' = —i, among the factors, only U{Li,si) acts non trivially on 0a, 
thus the morphism 



must be tight. Lemma 3.13 implies that U{Li) is of tube type, so si has van- 
ishing signature. Lemma 8.4 shows that the signature of A'^(I^) is ±dim(/£)/2, 
which does not vanish, contradiction. We conclude that if 2i is a root, it 




The signature of the subspace S'^V* of symmetric forms is 




U{Le)xU{Lg,)^U{Hom{Le,Le')) 



U{L,) ^ C/(A^(I,)) 



does not belong to =bP. 
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Proposition 8.6. Let G be a real form of 0'^{n,C). Let (f) : T ^ G he a 

reductive homomorphism. Let c C Q be the center of its centralizer. Then 
c is balanced with respect to (j) unless e = 1, dim(c) = 1, c* is generated 
by a root i with a 1- dimensional root space li, the sesquilinear form on Iq 
has vanishing signature, Gj± is reductive Hermitian of tube type and tightly 
embedded in U{Iq), and the homomorphism T —?■ U{Iq) is maximal. 

Proof: Assume that c is not balanced. From Lemma 8.1, we know that 
roots have to be pure imaginary. We take their imaginary parts without 
mentioning it explicitely. 

If e = —1 or if all le, i G L, have dimension > 1, then all 2i are roots and 
do not belong to itP, so they all belong to the complement N oi P (Lemma 
8.5). Since they generate c*, c is balanced, contradiction. So e = 1 and the 
set D of roots i (z L such that dim{Ie) = 1 is nonempty. 

Assume that D has at least 2 elements. According to Lemma 8.5, com- 
binations ±i lb i' , for i, i' G D, belong to N. Since they span span(Z)), 
and multiples 2i" , i" ^ D span span(L \ D), N spans c* and c is balanced, 
contradiction. So D has exactly one element, denoted by Iq. 

Assume that L ^ D. For £ ^ L», 2£ ^ ±P, hence 21 € N. Since 
4 U {£ ^ D} are all roots, dim(cVspan(A^)) = 1, and for aU ^ G L \ L», aU 
combinations it^o i ^ belong to itP. Since does not belong to the convex 
hull of the image of P in c*/span(A^), this convex hull contains exactly 
one of Iq and — ^o, say Iq. Then (.q — (. and (.q + (■ belong to P. Thus the 
sesquilinear representation of F in Hom{Iig , IiQl-i) is maximal. Lemma 3.6 
allows to replace Hom(Ii^^,Ii(Bl-.i) with Ii(Bl-i- However, as a sesquilinear 
vectorspace, is isomorphic to /_£, so, with Lemma 3.7, 

T{Ie © = T{Ie) + r(/„,) = r(/,) - r(/,) = 0, 

contradicting maximality. So L = D and dim(c) = 1. 

Assume that is not a root of c on C". Then dim(y = I^^ © I-io) = 2 
and Zg{c) = U{Iiq) = ^7(1), which cannot have any maximality property. 
So is a root on C", ±£0 are roots of c in the adjoint representation. Since 
does not belong to the convex hull of P in c*, exactly one of ±^0 belongs to 
P, say £0 € P. Then Hom{l£fj, Iq) is a maximal sesquilinear representation 
of F, so does /q, by Lemma 3.6. In particular, the signature of (/o,so) 
vanishes, i.e. U{Io) has tube type. Also, the morphism Zg{c) — >■ PU{Io) is 
tight. Zg{c) = Gj± X U{Ii^) acts on /q via the group Gj±, so the injection 
Gj± PU{Iq) is tight. This implies that Gj± is reductive Hermitian and 
has tube type (Lemma 3.13). ■ 

Corollary 8.7. Let G he a real form of 0^{n,C). Let (f) : T ^ G he a 

reductive homomorphism. Let c C Q be the center of its centralizer. Then c 
is balanced with respect to cj) unless G = S0*{2n), n odd, 0(F) C SO*{2n — 
2) X 50* (2) and (p is maximal. 
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Proof: In view of Proposition 8.6, there merely remains to determine which 
pairs (G, Gj± ) can lead to unbalanced centralizers, when G is a real form of 
0{N, C), i.e! G = 0{p, q) ov G = SO*{2n). 

In the real case, Iq is real, Gj-± is a real orthogonal group 0{p,q), which 
tightly injects into U{Io,so) = U{p,q). We also know that the signature 
p — g of So vanishes. 0{p,p) is reductive Hermitian only if p = 2. But 
Lemma 3.15 states that the inclusion 0(2, 2) ^ U (2, 2) is not tight. So c is 
always balanced if G = 0{p,q) is a real orthogonal group. 

In the quaternionic case, Iq is quaternionic and carries a nondegenerate 
( ~ , — l)-binary form (Lemma 4.1) (see subsection 4.2 also). Therefore Gj± 
is a quaternionic skew- unitary group S0*{2p), p = dime(/o)- Let £ denote 
the unique nonzero root of c on C^". 1^=1^® I_£ is a 2-dimensional com- 
plex vectorspace, thus a 1-dimensional quaternionic vectorspace, therefore 
n = p + 1. Since Gj± has tube type, p is even and n is odd. The homo- 
morphism F U{Iq) = U{p,p) is maximal. According to Lemma 10.3, the 



spaces Qp and Gp+i corresponding to Gj± and G have equal ranks and the 
embedding Qp ^ Qp+i is isometric and holomorphic. Therefore Example 
3.4 implies that, viewed as a homomorphism F — > G, is maximal as well. 



For homomorphisms F ^ G with reductive Zariski closure, the proof of 
Theorem 1 follows from Theorem 2.2, the classification of classical simple 
Lie groups and the case by case analysis of balancedness in sections 7 and 



Here is how the problem is reduced to the case of reductive homomor- 
phisms. Theorem 2 of [12] asserts that if genus(F) > 2dim(G)^ and G is 
semisimple, the space Hom{T,G) falls into two types of connected compo- 
nents: in some of them, Zariski dense homomorphisms are dense; others 
do not contain any Zariski dense homomorphism (call them rigid). Let 
(/) : F — 7- G be an arbitrary homomorphism. If cj) cannot be approximated 
by Zariski dense homomorphisms, then (p belongs to a rigid component C. 
Proposition 8.3 of [12] asserts that C contains a reductive homomorphism if^. 
The reductive case of Theorem 1 implies that G is Hermitian of non tube 
type and ^ is maximal. Since Toledo invariants are constant on connected 
components of HomiV, G), (j) is maximal as well. 



For the reader's convenience, we give barehanded proofs of the maximality 
preserving property of two embeddings between reductive Hermitian groups. 
We start with a preliminary observation. 




The symmetric 



9. Proof of Theorem 1 
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Lemma 10.1. Let F : Y ^ X be an equivariant totally geodesic map 
between Hermitian symmetric spaces. Assume Y is irreducible. Let P C X 
and Q CY be maximal polydisks such that F{Q) C P. Then F is positively 
maximality preserving if and only F\q : Q —?■ P is. 

Proof: Since Y is irreducible, F is homothetic, i.e. there exists a constant 
c such that F*ujx = cooy- Sectional curvature achieves its minimum along 
maximal poly discs, so 

{^x)\p = ^p, (^y)|Q = wq. 

Since rank(P) = lank(X) and rank(Q) = rank(y), F is positively maximal- 
ity preserving if and only if 

_ rank(X) _ rank(P) 

rank(y) rank(Q) 

,^ rank(P) 

if and only if F^q is positively maximality preserving. ■ 

Lemma 10.2. The embedding SU{n,n) ^ 5p(4n,R), is positively maxi- 
mality preserving. 

Proof: Let Y = T)n,n (resp. X = 821%) denote the symmetric space asso- 
ciated to H = SU{n,n) (resp. G = Sp{4n,R)). Let l : Y ^ X denote the 
corresponding embedding of symmetric spaces. We must show that 

L*UJX = 2UJY- 

Let us first study the case when n = 1. Let Vc = be equipped with 

the standard symmetric sesquilinear form v ■ v' = v~^v' . Let S" = ^ ^ 

The symmetric sesquilinear form s{v,v') = v ■ {Sv') on Vc has vanishing 
signature. It is easy to show that s{Av,Av') = s{v,v') for v,v' € C^, and 
A € 5L(2,M) by a direct calculation. Hence the group H = SU{Vc,s) 
coincides with S'L(2,R) acting on Vc = CS) C. Its maximal compact 



subgroup L is generated by J = ^ , 



Let Vk denote Vc viewed as a real vectorspace equipped with the sym- 
plectic form il{v,v') = Qm{s{v,v')). Then H = SU{Vc,s) is a subgroup of 
the larger symplectic group G = Sp{Vm.,^). Let p : SU{Vc,,s) ^ Sp{Vm,,^) 
denote the inclusion homomorphism. J' = p{J) is the 4x4 matrix which 

reads (y^^ in 2 x 2 blocks. J' is a complex structure compatible with 

O and tamed by $7, thus its centralizer in Sp{V^., Q) is a maximal compact 
subgroup K of SpiV^.., The adjoint actions of J on f)/[ and of J' on g/fi 
define the complex structures of the symmetric spaces Y and X associated 
to LI and G, so the /O-equivariant embedding l -.Y ^ X mapping L into K 
is holomorphic. 
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Let US view Vc as C. Then = © In this coordinates, 
SL(2, M) acts diagonally. The stabilizer of this decomposition in G is the 
standard 5L(2,M) x SL{2,R) of 5p(4,M). By Example 3.5, the embedding 
of symmetric spaces Y = Si ^ S2 corresponding to each 5L(2,M) ^ G 
is isometric and holomorphic, so SL{2,M) x SL{2,M) G gives rise to 
an isometric and holomorphic map of 1" x y onto a maximal polydisk P of 
X = 82. The image p{H) sits diagonally in the product SL{2, M) x SL{2, M), 
so t : y — )■ P factors through the diagonal A '.Y xY. This shows that 
Kahler forms fit up to a factor of 2, i.e. 

L*u}p = A*{prlujp +pr2U)p) = 2ujy- 

In general, let (Vc, s) be the orthogonal direct sum of n copies of the n = 1 
example just studied. Then s has vanishing signature. Each factor gives rise 
to a homomorphism SU{1,1) ^ H = 5f7(Vc,s) and a map T>i^i = Si ^ 
Y = 'Dn,n which, according to Example 3.3, is isometric and holomorphic. 
The product map 'Dfi ^ y is isometric and holomorphic onto a maximal 
polydisk Q of y. 

Let Vm. he Vc viewed as a real vectorspace equipped with the symplec- 
tic structure = 9m(s). Each factor of Vq is the complexification of a 
real 2-dimensional vectorspace. This gives rise to commuting embeddings 
SL{2,W) ^ Sp{An,W) and the corresponding map 5^" X = S2n is an 
holomorphic isometry onto a maximal polydisk P of X. The restriction of 
L :Y ^ X to Q is the direct product of n copies of the n = 1 case, so again 

L*UJp = 2UQ. 

With Lemma 10.1, since rank(X) = 2rank(y), this shows that i is positively 
maximality preserving. ■ 

Lemma 10.3. Let h he a nondegenerate (~ , skew) -symmetric binary form 
on a 2n- dimensional quaternionic vectorspace Vu (see subsection 4-1)- Use 
right multiplication by i to turn Vm into a complex An- dimensional vec- 
torspace Vc- LetC{q) = a denote the complex part of a quaternion q = a+jb. 
Then 

s{v, v') = C{h{v, v')) 

is a nondegenerate sesquilinear form of vanishing signature on Vq. Then the 
corresponding embedding of groups p : SO*{An) := 5C/(Vh,/i) SU{2n,2n) 
:= SUiVc) is positively maximality preserving. 

Proof: Let i -.Y = Q2n X = X^2n,2n be the corresponding embedding of 
Hermitian symmetric spaces. Since rank(^2n) = n and rank(I>2n,2n) = 2n, 
we shall show that Kahler forms match up to a factor of 2, i.e. i*ijJx = 
2ojY- Following Lemma 10.1, it suffices to understand the restriction of l to 
maximal poly disks. 

Let V ■ v' = v' denote the standard positive definite ~ -symmetric bi- 
nary form on H^". Let h{v,v') = v ■ iv'. Then h is {~ ,skew)-symmetric 
and nondegenerate, so we can take Vm = (IHI^",/i). The embedding p : 
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S'0*(4n) ^ SU{2n,2n) consists in taking a quaternionic matrix X, split- 
ting it as X = M + jM' where M and M' have complex entries, letting X 
act on the quaternionic vector v = a + jb, (a, b) G (C^")^ = C^". Thus 



Xv = {M + jM')ia + jb) = Ma- M'b + j{M'a + M6), 



I.e. 



(1) 



M -M' 
M' M 



Let J £ Gl{2n,M) denote left multiplication by i. Elements of Sp{2n) 
which commute with J (i.e. matrices with entries in C C H) form a group 
L isomorphic to U{2n). It is a maximal compact subgroup in SO* (An). 
Under p, this subgroup is mapped to the maximal compact subgroup K = 
S{U{2n) X U{2n)) hy M ^ {M,M). J belongs to the Lie algebra 1} = 
so*(4n), it generates the center of the Lie algebra [ of L. Therefore the 
complex structure on Q2n arises from the adjoint action of J on f)/[. For the 
same reason, the complex structure on T>2n,2n arises from the adjoint action 
of J' = diag{i, ■ ■ ■ ,i, —i, . . . , —i) on g/t. Note that at the Lie algebra level 
p{J) = J', thus L is holomorphic. 

Let us first study the case when n = 1. The Lie algebra 



so*(4) = {A\A* 



+ 



i 
i 



A = 0} 



is isomorphic to su(l, 1) ©su(2), it consists of matrices of the form 



la jx 
-jx ia 



+ 



ip 

-y 



y 

-03 



where a, /3 G M, x, y € C. The first matrix belongs to a subalgebra q 
isomorphic to su(l, 1). A computation based on formula (1) gives 



la 
-jx 



JX 

ia 



( ia 







ia 

X 







X 

-ia 




—X \ 



-ia 



We see that /9(q) is contained in the subalgebra 



/ ia 








— X \ 





ioL 


—x' 








—x' 


—ia' 





\-x 








—ia j 



; a, a' G M, x, x G C} 



which is isomorphic to su(l, 1) su(l, 1) and embedded in the standard 
(block diagonal) manner in su(2,2). The map /j|q : q — )■ p is the graph of 
an inner automorphism of su(l, 1). Geometrically, this means that l maps 
Q = (?2 (a complex line of constant curvature —1) holomorphically into a 
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maximal polydisk P of 1^2,2 ■ P is holomorphically isometric to Q x Q and 
''IQ • Q ~^ is the graph of an isometry / of Q. It follows that 

i*ujp = {id, I)* {prlujQ +pr2UjQ) = 2ujq. 

Let us map 50* (4) as a diagonal 2x2 block in S0*(4n). This yields 
an embedding Q2 ^ Q2n = Y which, according to Example 3.4, is isometric 
and holomorphic. Splitting T% = H^" as an orthogonal direct sum of 2- 
dimensional quaternionic vectorspaces yields an isometric and holomorphic 
embedding of Q2 onto a maximal polydisk Q C Q2n- Then i{Q) is contained 
in the standard maximal polydisk P C 'D2n,2n = X, and i\Q : Q ^ P \s a, 
product of n copies of the map of the previous paragraph. Therefore 

n n 

t^cjp = i*ujp. = ^WQ, = 2ujQ. 

i=l i=l 

Since rank(X) = 2rank(y), this shows that t (and thus p) is positively 
maximality preserving. ■ 
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